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ABSTRACT 

Making  an  evacuation  decision  in  response  to  a  hurricane  is  a  high-impact  event  that  directly  affects  public  safety  on 
a  large-scale.  Despite  being  extremely  critical  events,  the  frequency  of  hurricanes  is  low,  and  as  a  result  emergency 
managers  are  rarely  afforded  the  opportunity  to  engage  in  the  evacuation  decision  making  process,  real  or  simulated. 
Emergency  managers  are  currently  offered  in-person  courses  or  webinars  to  gain  an  introductory  understanding  of  the 
meteorological  and  evacuation  clearance  time  information  and  the  associated  tools  available  to  make  these  critical 
decisions.  Additionally,  the  offering  of  these  training  opportunities  is  infrequent,  limited  in  capacity  and  the  current 
training  techniques,  whether  in-person  or  by  webinar,  do  not  allow  emergency  managers  to  practice  critical  decision 
making,  or  measure  an  emergency  manager’s  ability  to  learn  the  information  correctly  and  then  transfer  it  to  an  actual 
evacuation  event. 

Through  this  work,  a  web-based,  ‘serious  gaming’  approach  was  used  to  develop  hurricane  evacuation  decision 
training  for  the  emergency  manager.  This  paper  describes  the  iterative  design  approach  to  developing  a  training  game 
and  collect  initial  feedback  from  emergency  managers  on  the  prototype  capability.  Instructional  goals  were  identified 
in  conjunction  with  the  Department  of  Homeland  Security  Science  and  Technology  Directorate  as  well  as  the  Federal 
Emergency  Management  Agency  (FEMA)  and  the  National  Hurricane  Center  (NHC),  who  jointly  conduct  the  current 
hurricane  evacuation  decision  training  on  an  annual  basis.  The  game  platform  was  integrated  with  the  recently  re¬ 
designed  hurricane  evacuation  decision  support  tool  (HVX)  and  has  the  advantage  of  reaching  many  more  state  and 
local  emergency  managers  without  time  or  funds  to  attend  the  in-person  training.  Furthermore,  the  embedded 
analytical  capabilities  enable  emergency  managers  to  identify  difficult  concepts  and  allow  them  to  further  practice 
these  concepts  within  simulated  hurricane  scenarios  until  a  performance  criterion  is  achieved. 
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INTRODUCTION 

Hurricanes  can  be  devastating  events.  Rappaport  (2014)  published  that  over  2500  deaths  were  due  to  Atlantic  tropical 
cyclones  over  the  past  50  years.  One  of  the  key  ways  that  deaths  are  prevented  in  hurricanes  is  removing  those  who 
would  be  affected  through  the  evacuation  process.  The  responsibility  for  evaluating  the  need  for  a  hurricane 
evacuation  falls  to  the  state  and  local  emergency  managers.  Depending  on  the  locality,  emergency  managers  may  call 
for  the  evacuation  themselves  or  advise  the  local  elected  official  to  do  so. 

The  evacuation  decision  is  not  a  simple  decision,  and  it  is  fraught  with  uncertainty.  A  major  source  of  uncertainty  is 
the  weather  forecasts.  At  three  days  from  an  impending  hurricane,  the  forecast  uncertainty  can  mean  the  difference 
between  no  impacts  on  the  community  or  complete  devastation.  Another  evacuation  consideration  is  how  long  an 
area  takes  to  evacuate,  which  can  be  determined  by  elements  such  as  geography,  road  structure,  and  population  density. 
Over  the  last  25  years,  emergency  managers  have  had  the  hurricane  evacuation  tool,  more  commonly  known  as 
HURREVAC,  to  help  them  determine  the  appropriate  time  to  evacuate,  according  to  the  weather  forecasts  and 
evacuation  requirements. 

Development  and  maintenance  of  the  HURREVAC  tool  is  supported  by  the  National  Hurricane  Program  (NHP), 
which  is  a  joint  effort  between  the  Eederal  Emergency  Management  Agency  (FEMA),  the  National  Hurricane  Center 
(NHC)  and  the  Army  Corps  of  Engineers.  In  addition  to  support  for  the  tool  itself,  NHP  also  trains  emergency 
managers  on  how  to  use  HURREVAC  and  hurricane  preparation  and  response  decision  making  as  well. 

In  2014,  the  Department  of  Homeland  Security  in  coordination  with  FEMA  conducted  a  study  of  the  HURREVAC 
tool  with  respect  to  modernizing  the  technology  (MITLL/SNL,  2014).  In  this  study,  over  50  emergency  managers 
from  across  the  United  States  were  interviewed  to  identify  the  critical  needs  of  the  emergency  managers  for  hurricane 
decision  making.  Specific  investigation  of  the  state  of  the  HURREVAC  tool  was  conducted,  including  a  software 
architecture  evaluation  and  emergency  manager  needs  gap  assessment.  One  result  of  this  study  determined  that  a 
software  update  of  the  HURREVAC  tool  was  necessary  and  additional  emergency  manager  decision  support  tools 
were  desired.  The  study  also  revealed  a  need  for  more  emergency  manager  training  opportunities  in  hurricane 
evacuation  decision  making  and  associated  training  tools.  This  HURREVAC  update  is  currently  in  the  process  of 
development,  and  the  updated  tool  is  called  HVX  (HURREV AC-extended).  HVX  will  address  the  shortcomings  of 
the  HURREVAC  tool  and  also  provide  integrated  training  opportunities. 

National  Hurricane  Program  Training  Assessment 

The  current  training  available  from  the  National  Hurricane  Program  are  live  instructor  courses,  webinars,  and 
HURREVAC  help  text.  The  live  instructor  courses  are  offered  in  a  one  and  five  day  format.  The  five-day  hurricane 
decision  making  course  (FEMA  L-324)  covers  four  days  of  education  on  the  National  Hurricane  Center  weather 
products  and  one  day  of  HURREVAC  training.  On  the  HURREVAC  day,  one  or  two  hurricane  scenarios  are 
presented  to  approximately  20  students  and  they  are  walked  through  example  decision  making  using  the  simulated 
storm.  This  interaction  is  conducted  with  each  of  the  students  loading  the  simulated  storm  into  their  laptop’s 
HURREVAC  tool  and  following  along  as  a  group  with  the  instructor  as  he  creates  specific  reports  and  particular 
visualizations.  The  one  day  course  consolidates  the  information  from  the  five-day  course  into  a  single  day. 
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The  webinars  explain  the  features  of  HURREVAC  and  demonstrate  how  to  access  features  using  video-guided 
scenarios  that  answer  specific  questions  relevant  to  emergency  managers. 

The  HURREVAC  help  text  is  an  exhaustive  description  of  each  feature  in  HURREVAC  with  supporting  information 
on  how  to  best  use  this  information  and  when  NOT  to  use  it  as  well. 

During  the  NHP  Technology  Modernization  study,  expansion  of  training  opportunities  was  one  of  the  primary 
suggestions  provided  by  emergency  managers.  One  issue  that  emergency  managers  have  is  that  the  live  instruction 
classes  are  a  limited  offering  and  only  allow  a  handful  of  emergency  managers  to  be  trained  in  a  year.  These  classes 
are  often  full,  especially  the  five-day  course.  At  the  same  time,  travel  budgets  for  emergency  managers  are  limited, 
and  the  travel  required  for  the  live  courses  results  in  an  emergency  manager  receiving  the  training  only  once  or  twice 
in  his  or  her  career,  when  ideally  there  should  be  extended  initial  training  and  then  seasonal  refresher  training.  Because 
of  the  seasonal  and  infrequent  nature  of  the  hurricane  threat,  often  the  training  becomes  “stale”  in  the  mind  of 
emergency  managers  at  the  start  of  the  season  or  when  a  hurricane  is  imminent.  This  could  result  in  inappropriate  use 
of  information  for  hurricane  evacuation  decisions  or  confusion  when  the  hurricane  threat  arises.  Another  issue  that 
was  identified  by  the  Technology  Modernization  analysis  team  was  that  the  current  training  contains  limited 
assessment  techniques  to  determine  whether  the  emergency  managers  have  actually  learned  and  can  apply  the 
information  provided  in  the  training. 

NHP  Training  Needs 

The  research  showed  that  emergency  managers  are  in  need  of  training  from  NHP  in  three  key  areas:  how  to  access 
the  features  in  HVX  (e.g.,  the  “buttonology”),  initial  training  of  hurricane  evacuation  decision  making  for 
new/unfamiliar  emergency  managers,  and  refresher  training  of  decision  making  for  emergency  managers  at  the 
beginning  of  a  season  or  hurricane  event. 

The  NHP  should  provide  remote  training  opportunities  to  accommodate  emergency  managers  with  minimal  budgets. 
This  training  should  be  within  the  HVX  tool  itself  so  that  there  is  not  a  different  system  or  download  required.  The 
training  should  be  relevant  to  the  emergency  management  environment,  more  specifically,  the  training  questions 
should  use  actual  or  simulated  storms  with  questions  that  an  emergency  manager  would  ask  (Mautone,  et  al.,  2008). 
There  should  also  be  a  means  to  assess  whether  the  emergency  managers  are  learning  the  training  materials. 

Another  requirement  was  to  pursue  the  most  engaging  training  possible,  and  the  means  chosen  to  achieve  this 
engagement  is  the  serious  gaming  method.  Evidence  has  suggested  that  gaming  is  an  effective  means  of  conveying 
the  training  objectives  with  high  levels  of  engagement  (Garris,  et  al.,  2002). 

Eor  the  remainder  of  this  paper,  an  Instructional  System  Design  method  was  used  to  convey  the  information  on  design, 
development  and  assessment  of  the  NHP  training  system  prototype  (Merrill,  et  al.,  1996). 


DESIGNING  AN  EVACUATION  DECISION  MAKING  TRAINING  MODULE 

The  design  of  the  evacuation  decision  making  training  module  was  completed  with  the  aid  of  the  current  NHP  training 
team  from  FEMA,  NHC,  and  Army  Corps  of  Engineers.  The  group  held  several  multi-hour  sessions  to  discuss  and 
agree  upon  the  content  of  the  module.  The  content  was  also  presented  to  and  agreed  upon  with  the  greater  membership 
of  the  NHP. 

Instructional  Goal 

The  instructional  goal  agreed  upon  for  the  training  module  was,  “Understanding  which  NHC  products  are  used  for 
which  decision  making  timeframe.” 

Instructional  Objectives 
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To  identify  the  objectives  required  to  achieve  the  instructional  goal,  a  matrix  of  the  decision-making  timeframes  and 
the  emergency  management  tasks  was  generated,  as  shown  in  Figure  1  below. 
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Figure  1:  Instructional  objectives  matrix  for  the  NHP  training  module. 

The  five  decision  making  timeframes  were:  1)  Before  storm  formation,  2)  Early  storm  awareness  (Five-t  days  until 
storm  impacts),  3)  Pre-watch/warning  storm  awareness  (two  to  five  days  until  storm  impacts),  4)  W atch/warning  phase 
(0-48  hours  until  storm  impacts),  and  5)  Monitor  storm  during  impact  &  initial  recovery  phase  (after  storm  impacts 
have  begun).  The  three  emergency  management  tasks  the  training  module  would  focus  on  are  storm  situation 
awareness,  impact/risk  assessment,  and  evacuation  decision  making.  For  each  of  the  cells  of  tasks  and  decision 
making  timeframes,  the  training  team  identified  which  products  were  available/applicable,  how  to  access  these 
products,  which  products  used  together  can  answer  the  emergency  manager’s  questions,  and  how  to  avoid  common 
missteps. 

Instructional  Strategies 

The  strategies  that  the  training  team  identified  to  accomplish  these  objectives  were  a  tutorial  and  assessment  questions. 
Tutorials  led  emergency  managers  through  how  to  access  the  NHC  products  and  what  information  these  products 
offered.  Then  scenario-based  assessment  questions  that  were  provided  required  the  emergency  managers  to  know 
when  to  access  these  products,  how  to  interpret  them,  and  what  the  limitations  of  the  products  are.  “Level  1” 
assessment  questions  were  simple,  straightforward  questions  designed  to  build  confidence  of  emergency  managers 
who  read  and  understood  the  information  provided  in  the  tutorials.  “Level  2”  assessment  questions  were  then  provided 
to  determine  whether  the  emergency  manager  understood  the  subtleties  of  the  information.  These  questions  are 
“trickier”  and  built  to  identify  the  common  misuse  of  the  products. 

Evaluation  Methods 

The  evaluation  method  identified  was  specific  to  the  application  of  the  NHC  products  in  a  given  context.  The  training 
team  wanted  to  ensure  that  the  focus  of  the  training  was  not  on  whether  the  evacuation  decision  was  “correct”  or 
“incorrect,”  but  rather  whether  the  NHC  products  used  to  make  those  decisions  were  used  at  the  correct  timeframe  in 
the  correct  context.  Therefore,  while  feedback  on  the  evacuation  decision  could  be  provided  (e.g.,  40  percent  of 
emergency  managers  who  took  this  training  chose  to  evacuate  h-18  hours  until  the  arrival  of  tropical  storm  force  winds, 
30  percent  chose  to  evacuate  h-19  hours.),  feedback  on  the  use  of  specific  products  was  provided  using  the  following 
ratings — appropriate,  inappropriate,  or  provides  no  information. 

Types  of  Training  Materials 

Once  the  specific  training  objectives  were  identified,  the  training  team  then  identified  past  storms  that  would  provide 
interesting  contexts  for  each  of  the  decision  timeframes.  The  specific  information  in  the  tutorials  and  the  specific 
wording  of  assessment  questions  were  also  identified. 


2016  Paper  No.  16313  Page  4  of  10 


For  public  release.  Distribution  A.  Approved  for  public  release:  unlimited  distribution. 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2016 


Media  Selection  (Serious  Game) 

As  previously  discussed,  the  media  for  training  delivery  was  a  serious  game  method.  The  age  range  of  the  emergency 
managers  to  be  trained  was  anywhere  from  20-70,  and  many  of  the  emergency  managers  with  decision-making 
responsibility  were  in  the  older  age  ranges.  The  serious  game  method  is  challenged  because  there  is  a  portion  of  the 
emergency  manager  population  that  is  not  comfortable  with  using  computers  and  may  not  be  interested  in  games  as  a 
training  method.  Efforts  must  be  made  during  development  to  evaluate  the  usability  of  the  training  once  it  is  developed 
to  ensure  it  is  feasible  across  the  emergency  manager  community. 

Design  Document 

For  the  game  development,  the  information  about  the  products,  objectives,  storms,  and  questions  were  input  into  a 
spreadsheet  to  allow  the  developers  to  produce  a  gamed  approach  to  the  training. 


DEVELOPMENT  OF  A  SERIOUS  GAME  INFRASTRUCTURE  FOR  TRAINING 

This  section  describes  the  concept  of  use  for  the  NHP  training  game.  The  game  itself  is  currently  mid-development, 
thus  some  functionality  is  not  yet  fully  developed.  The  currently  developed  prototype  is  described  in  the  next  section. 

There  was  a  requirement  by  the  emergency  managers  that  they  would  not  have  to  go  to  a  different  website  or  download 
different  software  to  accomplish  the  training  tasks.  Thus,  the  implementation  of  the  training  itself  is  a  mode  within 
the  overall  HVX  tool. 

Training  mode  in  HVX  is  accessed  through  the  selection  of  a  “Training”  tab  at  the  top  left  of  the  HVX  tool.  To  enter 
the  HVX  tool,  the  emergency  manager  is  required  to  login  to  his  or  her  specific  profile.  This  user  profile  not  only 
saves  the  emergency  manager’s  individual  configuration  settings  and  favorite  locations  for  the  HVX  tool,  but  it  also 
contains  their  training  data.  This  data  includes  what  modules  have  been  completed  and  when,  what  their  performance 
was,  and  what  HVX  features  were  accessed  to  answer  each  question.  Ultimately,  this  data  can  be  viewed  by  the 
trainee  as  well  as  their  trainers  with  the  appropriate  permissions. 


Developing  Course  Materials 

The  learning  goals,  specific  learning  objectives,  and  example  storms  were  identified  in  conjunction  with  the  NHP 
training  team,  described  in  the  previous  section.  These  goals,  objectives,  and  storms  were  inserted  into  the  matrix 
shown  in  Figure  1  as  the  game  objectives  matrix.  For  each  cell  in  the  matrix,  a  tutorial  walkthrough  or  specific 
question  was  generated  to  accomplish  the  objective.  The  final  state  of  the  course  materials  resulted  in  a  series  of 
scripted  tutorials  followed  by  learning  assessments  for  each  HVX  module. 

Evaluation  Instruments 

There  are  several  means  to  evaluate  the  emergency  manager  trainee  within  the  training.  One  means  of  assessing  the 
trainee  are  the  answers  to  the  gaming  assessments.  These  “answers”  can  be  tagged  as  “correct”  or  “incorrect.” 
Alternatively,  some  questions  asked  in  the  gaming  have  no  “right”  answer.  The  evacuation  decision  is  an  example  of 
this  type  of  question.  Depending  on  the  local  standard  operating  procedures  and  each  emergency  manager’s  tolerance 
for  risk,  the  ultimate  evacuation  decision  can  be  personal  and  is  too  complex  to  be  determined  “correct”  or  “incorrect” 
in  an  absolute  sense.  The  trainee’s  answer  for  this  type  of  question  is  presented  back  to  them  with  reference  to  all  of 
the  answers  that  other  trainees  have  provided  (e.g.,  40  percent  answered  h-18  hours  and  60  percent  answered  h-19 
hours).  For  this  type  of  question,  the  assessed  element  would  be  HOW  the  trainee  got  to  that  answer.  In  this  case,  the 
“correct”  answer  would  be  that  the  trainee  used  a  particular  feature  of  HVX.  (For  example,  “correct”  in  a  question 
could  be  that  they  accessed  the  wind  radii  features,  and  “incorrect”  could  be  that  they  accessed  the  wind  swath  for  that 
question.) 
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Assessments  are  conducted  after  each  question,  as  described  above,  and  they  are  also  provided  at  the  end  of  the 
module.  Performance  for  a  module  is  provided  by  giving  percentage  of  “correct”  and  “incorrect”  answers,  but  also 
how  many  times  a  particular  HVX  feature  was  used  “correctly”  or  “incorrectly.” 


Game  Infrastructure 

The  NHP  training  game  was  created  using  a  game  engine  written  in  JavaScript.  It  uses  an  HTML5  canvas  layered  on 
top  of  the  HVX  interface  in  order  to  display  the  state  of  the  game.  The  state  of  the  game  is  described  using  two  sets; 
a  set  of  entities  and  a  set  of  components.  An  entity  is  something  that  can  be  acted  upon  and  has  a  set  of  properties. 
Components  are  a  single  behavior,  such  as  the  drawing  of  an  entity  on  the  screen  or  the  parsing  of  player  input,  which 
act  according  to  the  properties  of  the  entity  using  the  component.  These  two  first-order  types  are  used  to  construct  a 
table,  mapping  entities  to  the  components  they  use.  Once  an  initial  game  state  is  constructed,  every  frame,  1/60*  of  a 
second,  the  game  state  is  updated.  The  game  state  is  updated  by  executing  the  behavior  of  each  component  on  all 
entities  in  the  game  state  using  the  component.  During  the  course  of  play,  any  entity  may  be  marked  as  using  or  not 
using  any  component  and  any  entity  can  be  removed  from  the  game  state  at  any  time  a  new  game  state  is  created. 


PROTOTYPE  GAME  IMPLEMENTATION  AND  EVALUATION 

To  evaluate  the  game  training  concept  with  emergency  managers  before  extensive  training  modules  and  capabilities 
were  designed  and  implemented,  a  prototype  training  game  was  developed.  In  this  prototype,  three  features  of  HVX 
were  chosen,  and  tutorials  and  assessment  scenarios  were  written.  The  prototype  training  module  was  implemented 
within  HVX,  and  the  assessment  questions  within  HVX  were  simplified  to  “correct”  or  ’’incorrect”  answers.  After 
answering  a  question,  the  trainee  received  feedback  about  whether  their  answer  was  “correct”  or  “incorrect,”  and  at 
the  end  of  the  prototype  module,  an  overall  score  was  provided. 

The  purpose  of  this  prototype  module  was  to  implement  initial  capabilities  for  the  entirety  of  a  training  capability,  and 
then  use  this  prototype  to  gain  feedback  from  emergency  managers  both  in  a  conceptual  sense  and  a  usability  sense. 
The  usability  portion  is  important  because  of  the  issue  of  technological  discomfort  within  the  emergency  manager 
population  and  the  remote-use  nature  of  the  training  game.  As  this  program  is  mid-development,  an  assessment  of 
both  the  training  concept  and  the  usability  could  provide  valuable  feedback  to  the  remainder  of  the  development  period 
and  aid  in  prioritizing  development  of  the  training  capabilities. 

Prototype  Game 

The  initial  screen  of  the  training  prototype  is  shown  in  Figure  2  below.  When  the  training  mode  tab  is  clicked  from 
HVX  (red  box),  the  introductory  information  box  appears. 


o  □ 


Figure  2:  Introductory  screen  to  the  NHP  training  game  prototype. 
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In  the  tutorial  segments  following  the  introductory  information  box,  trainees  are  guided  through  the  process  of 
selecting  an  archived  storm,  choosing  an  advisory  for  the  storm,  and  accessing  the  HVX  feature  (the  error  cone  as  the 
first  example).  To  aid  the  trainees  in  navigating  the  key  buttons,  a  “hint”  functionality  was  implemented.  An 
information  box  asks  the  trainee  to  access  the  feature  (e.g.,  “Click  on  the  Error  Cone.”)  and  it  also  provides  a  “Hint” 
button  at  the  bottom.  If  the  trainee  clicks  on  the  hint  button,  a  spotlight  zooms  in  on  the  button,  lowlighting  all  other 
information  so  that  it  is  easily  identified. 

Once  a  feature  is  accessed,  key  information  about  that  feature  is  conveyed  to  the  trainee  about  what  that  feature  is  for, 
how  it  should  be  used,  and  errors  that  are  commonly  made  with  that  feature.  Instead  of  a  series  of  information  boxes 
to  convey  this  information,  trainees  are  asked  to  click  on  different  parts  of  the  map  to  show  that  they  know  what  part 
of  that  feature  the  tutorial  is  describing.  This  enables  a  high  level  of  interactivity  instead  of  blindly  clicking  “Next” 
through  an  endless  series  of  information  boxes. 

After  the  information  is  conveyed  about  the  feature,  simple  scenario-based  questions  using  this  information  are  asked. 
These  Level  1  questions  are  simple  and  designed  to  reinforce  the  information  previously  presented.  Figure  3  shows 
an  example  Level  1  question  for  the  wind  radii  feature.  The  question  in  the  information  box  states,  “Based  upon  this 
advisory,  progress  the  forecast  hour  to  the  hour  when  tropical  storm  force  winds  are  likely  to  begin  to  impact  the  state 
of  North  Carolina.” 


Figure  3:  Level  1  training  scenario-based  training  question  for  the  wind  radii  feature. 


The  answer  for  game-based  training  is  generally  not  provided  in  a  multiple  choice  format,  but  rather  through 
interacting  with  the  features  in  the  tool.  In  this  case,  the  answer  is  provided  by  clicking  the  “next  forecast  hour”  button 
until  the  hour  in  which  the  tropical  storm  force  winds  enter  North  Carolina  and  then  the  trainee  presses  submit  when 
satisfied. 


Once  the  trainee  completes  the  tutorials  in  the  prototype,  the  trainee  is  prompted  with  a  set  of  Level  2  assessment 
questions.  In  this  prototype,  three  questions  are  provided,  each  of  which  covers  a  different  training  feature.  The  Level 
2  questions  are  slightly  more  challenging  than  the  Level  1  simple  questions,  and  they  are  not  preceded  by  a  tutorial  of 
the  feature.  Thus,  the  trainee  must  decide  which  feature  is  appropriate,  and  then  answer  a  challenging  question  about 
it.  Figure  4  shows  an  example  of  a  Level  2  question.  The  question  states,  “In  your  discussion  with  Virginia’s  governor 
during  Advisory  #26,  he  asks  if  any  cities  have  a  low  probability  of  experiencing  wind  impacts.  Which  of  these 
Virginia  cities  has  a  less  than  30  percent  probability  of  experiencing  tropical  storm  force  winds  over  the  next  five 
days?  Select  the  applicable  location  denoted  by  pins  and  submit  your  answer.”  The  trainee  can  then  click  the  pin  of 
the  answer  and  press  the  “Submit”  button. 
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Figure  4:  Level  2  assessment  scenario  question  for  the  wind  probability  graphic. 


Training  Evaluation  of  Prototype  Game 

An  initial  training  evaluation  was  conducted  at  the  Florida  Governor’s  Hurricane  Conference,  a  venue  at  which  many 
emergency  managers  receive  training  and  share  information  about  hurricane  weather  products  and  procedures.  This 
evaluation  had  three  purposes:  1)  to  allow  emergency  managers  to  experience  the  training  concept  in  a  hands-on 
format,  2)  to  evaluate  the  concept  of  the  remote  training  platform  in  its  current  instantiation,  and  3)  to  gain  insight  into 
any  outstanding  usability  issues  that  might  hinder  the  training  process. 

Four  Florida  emergency  managers  from  different  counties  volunteered  their  time  to  participate  in  the  evaluation  during 
the  conference.  The  average  experience  of  the  emergency  managers  with  HURREVAC  was  8.75  years  (two  years 
minimum,  17  years  maximum).  While  four  data  points  would  not  provide  statistically  significant  results,  the 
qualitative  outcomes  met  the  goals  of  the  evaluation. 

The  emergency  manager  volunteers  were  briefly  introduced  to  HVX  and  the  purpose  of  the  training,  and  then  they 
were  asked  to  complete  the  prototype  training  module,  which  took  approximately  15  minutes  to  complete.  During  the 
training  module,  two  researchers  observed  their  interactions  and  took  note  of  their  comments.  At  the  end  of  the 
training  module,  they  were  asked  to  complete  a  short  survey  about  the  training  prototype  and  the  training  concept  as 
a  whole. 

In  the  surveys,  a  question  was  asked  and  then  a  Likert  rating  scale  was  provided  from  1  (not  at  all)  to  7  (very).  The 
following  questions  were  asked  in  the  survey  about  the  current  implementation  of  the  training  prototype: 


Question 

Average  rating 

Minimum/Maximum  rating 

How  useful  would  the  style  of  training  demonstrated  in 
the  prototype  be  to  emergency  managers  who  use 

HVX? 

6.25 

5/7 

How  engaging  was  the  training  demonstrated  in  the 
prototype? 

6 

4/7 

How  useful  was  the  HVX  feature  tutorial? 

5.75 

4/7 

How  useful  were  the  scenario-based  questions  that 
followed  the  tutorials? 

6.75 

6/7 
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For  the  instantiation  of  the  NHP  training  game  in  the  prototype,  overall,  the  average  usefulness  and  engagement  scores 
were  high.  The  feature  tutorial  and  the  engagement  level  were  the  items  that  appeared  to  need  the  most  work.  The 
emergency  manager  volunteers  saw  value  in  the  style  of  training  and  the  scenario-based  questions.  The  write-in 
comments  on  this  section  of  the  survey  had  suggestions  to  improve  engagement  such  as  adding  humor  into  the  delivery 
of  the  materials  and  using  as  many  real-world  scenarios  as  possible.  Verbally,  one  volunteer  suggested  that  this 
training  could  follow  the  Red  Cross  CPR  online  training,  which  he  thought  was  very  engaging  and  effective. 

The  survey  also  asked  questions  about  the  training  game  concept  in  general: 


Question 

Average  rating 

Minimum/Maximum  rating 

How  useful  would  a  web-based  training  capability  be? 

6.75 

6/7 

How  useful  would  a  training  capability  that  is  within 
HVX  and  using  HVX  be? 

7 

7/7 

How  likely  are  you  to  recommend  a  web-based 
training  that  teaches  the  HVX  buttons  and  features  to 
new  emergency  managers? 

6.75 

6/7 

How  likely  are  you  to  recommend  a  web-based 
training  that  teaches  emergency  managers  techniques 
for  effective  use  of  the  National  Hurricane  Center 
products  for  decision  making? 

6.25 

4/7 

How  useful  would  it  be  to  see  the  consequences  of 
various  evacuation  decisions  to  different  stakeholders 
(e.g.,  elected  officials,  local  tourism  business)  in  a 
training  capability? 

6.5 

5/7 

There  appeared  to  be  positive  feedback  about  a  web-based  training  capability  in  general.  There  was  consensus  that 
web  training  covering  the  HVX  features — the  “buttonology” — would  be  valuable.  However,  one  emergency 
managers  thought  that  the  decision  making  use  and  the  stakeholder  feedback  concept  were  “good  ideas,  but  beyond 
HVX  scope.” 

During  the  observation  of  the  training  module  completion,  several  usability  issues  were  observed.  The  information 
boxes  in  which  the  training  information  was  conveyed  were  not  movable  and  only  white.  Two  volunteers  tried  to 
move  these  boxes  to  other  locations,  and  one  volunteer  suggested  that  the  color  of  the  box  relate  to  what  type  of  action 
was  expected  (e.g.,  push  “next”  button,  select  a  forecast  hour,  click  a  pin).  Some  map  features  required  several  clicks 
to  register  as  “selected,”  which  was  unacceptable  and  confusing  to  several  volunteers.  Two  volunteers  also  found 
that  the  clickable  pin  area  was  too  small  to  be  adequately  usable. 

The  training  performance  was  also  assessed.  For  the  Level  1  questions  that  followed  each  tutorial,  the  four  volunteers 
scored  100  percent  correct.  This  was  as  expected  with  the  way  the  module  was  designed,  because  Level  1  was 
expected  to  be  simple  and  straightforward.  The  Level  2  questions  were  designed  to  be  more  difficult.  75  percent  of 
the  volunteers  got  the  error  cone  question  incorrect.  50  percent  got  the  wind  radii  question  incorrect.  25  percent  got 
the  wind  probability  graphic  question  incorrect.  The  expectation  for  the  trainee  is  that  they  would  then  review  the 
tutorial  portion  on  the  features  for  the  questions  that  were  answered  incorrectly  to  gain  insight  into  how  they  could 
improve  their  performance.  The  responses  aligned  well  with  the  intended  design  for  the  questions,  even  if  there  were 
only  four  volunteers  for  the  evaluation. 


DISCUSSION  AND  NEXT  STEPS 

Overall  the  NHP  training  game  prototype  was  well-received  by  the  limited  volunteers  in  the  training  evaluation. 
Informally,  the  concept  has  been  presented  to  the  larger  HVX  users  group  consisting  of  20+  emergency  managers, 
FEMA  representatives,  NHC  representatives  and  other  interested  parties.  Positive  feedback  has  been  received  at  these 
meetings  and  has  corroborated  the  findings  within  this  report. 
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The  next  step  in  NHP  training  game  development  is  expanding  the  prototype  training  module  to  a  full,  complete 
module  with  many  more  features  and  decision  timeframe  questions.  One  major  developmental  focus  will  be 
improving  the  engagement  quality  of  the  game.  Ideas  for  how  to  do  this  were  presented  at  the  most  recent  user  group 
meeting.  One  idea  involved  including  stakeholder  advisors  who  would  provide  their  input  on  a  trainee’s  decisions. 
The  input  could  be  provided  in  a  casual  manner,  possibly  improving  the  humor  in  the  game  as  well.  The  other 
developmental  priority  is  improving  the  usability  of  the  training  game.  Because  of  the  trainee  audience,  ensuring  that 
the  training  system  is  easy  to  use  and  not  confusing  is  a  critical  component  to  the  success  of  the  training  game. 
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